Using call stack snapshots to detect anomalous computer behavior

ABSTRACT

Detecting computer anomalies by determining probabilities of encountering call stack configurations at various depths, the call stacks being associated with software application instances on computers having the same operating system, where snapshots of the call stacks are recorded on the computers responsive to detecting predefined software application events, determining entropies of call stack configurations at various call stack depths using their associated probabilities, determining stack frame rarity scores of call stack configurations at various depths based on their associated stack frame entropies in accordance with a predefined rarity function, determining a call stack rarity score of any given call stack configuration as the maximum stack frame rarity score of the given configuration, and detecting an anomaly associated with any given one of the computers where any of the snapshots recorded on the given computer is of a call stack whose call stack rarity score meets a predefined anomaly condition.

BACKGROUND

Malware exploits take advantage of bugs and flaws in existing software,and cause them to divert execution along a route dictated by the malwareauthor. For example, a malware author can exploit a bug in the renderingengine of a web page browser by creating a web page that is configuredto trigger the bug and thereby cause the browser to execute maliciouscode that is embedded in the delivered web page, such as in JavaScript™or even in its image resources.

SUMMARY

A common effect of malware exploits that take advantage of bugs andflaws in existing software is that the call stack of the software thatis the target of the exploit will reflect the path of the executedmalicious code. The invention, in embodiments thereof, detects anomalouscomputer behavior, such as may be caused by malware, by analyzinganomalies in call stack snapshots taken at various control points ofsoftware applications during their execution on multiple computers.

In one aspect of the invention a computer anomaly detection method isprovided, the method including determining stack frame probabilities ofencountering various configurations of multiple call stacks at variouscall stack depths, where the call stacks are associated with multipleinstances of a software application on multiple computers having thesame operating system, and where multiple snapshots of the call stacksare recorded on the computers responsive to detecting a predefined eventin connection with the software application, determining stack frameentropies of various configurations of the call stacks at various callstack depths based on their associated stack frame probabilities,determining stack frame rarity scores of various configurations of thecall stacks at various call stack depths based on their associated stackframe entropies in accordance with a predefined rarity function,determining a call stack rarity score of any given configuration of thecall stacks as the maximum stack frame rarity score of the givenconfiguration, and detecting an anomaly associated with any given one ofthe computers where any of the snapshots recorded on the given computeris of a call stack whose call stack rarity score meets a predefinedanomaly condition.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the invention will be understood and appreciated more fullyfrom the following detailed description taken in conjunction with theappended drawings in which:

FIG. 1 is a simplified conceptual illustration of a computer anomalydetection system, constructed and operative in accordance with anembodiment of the invention;

FIGS. 2A and 2B, taken together, is a simplified flowchart illustrationof an exemplary method of operation of the system of FIG. 1, operativein accordance with an embodiment of the invention; and

FIG. 3 is a simplified block diagram illustration of an exemplaryhardware implementation of a computing system, constructed and operativein accordance with an embodiment of the invention.

DETAILED DESCRIPTION

Embodiments of the invention may include a system, a method, and/or acomputer program product. The computer program product may include acomputer readable storage medium (or media) having computer readableprogram instructions thereon for causing a processor to carry outaspects of the invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the invention.

Aspects of the invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Reference is now made to FIG. 1, which is a simplified conceptualillustration of a computer anomaly detection system, constructed andoperative in accordance with an embodiment of the invention. In thesystem of FIG. 1, each computer in a group of computers 100 isconfigured with the same operating system 102 and a monitoring agent 104that monitors an instance of the same software application 106, such asof a browser, on the computer. On each computer in group 100, monitoringagent 104 preferably detects occurrences of one or more predefinedevents in connection with software application 106, such as createprocess events, which detection may be triggered using event-specifickernel hooks implemented in accordance with conventional techniques atone or more control points. When monitoring agent 104 detects any of thepredefined events, monitoring agent 104 records a snapshot 108 of a callstack 110 associated with software application 106, where snapshot 108includes a descriptor identifying the event, as well as informationregarding one or more stack frames S₀ . . . S_(n), preferably up to amaximum depth of consecutive stack frames, such as 10, where S₀ is thelast call before switching to kernel mode for the system call. Snapshot108 preferably includes, for each stack frame, the event type, themodule name, the module size, and address information, such as includingthe module base address and the call return address, and/or includingthe offset of the call return address relative to the module baseaddress. An example of an instance of a snapshot 108 is as follows:

{   “event”: {     “time”: “Oct 18, 2015 08:31:00”,     “name”:“process_create_event”,     “pid”: “1512”,     “0_frame”: “15919964”,    “0_ip”: “2089865658”,     “0_mod_base”: “2089811968”,    “0_mod_name”: “ntdll.dll”,     “0_mod_size”: “729088”,    “1_frame”: “15919968”,     “1_ip”: “2088865493”,     “1_mod_base”:“2088763392”,     “1_mod_name”: “kernel32.dll”,     “1_mod_size”:“1007616”,     “2_frame”: “15922744”,     “2_ip”: “2011081068”,    “2_mod_base”: “2010972160”,     “2_mod_name”: “advapi32.dll”,    “2_mod_size”: “634880”,     ...} ...}where #n_(—) indicates properties of stack frame n relative to azero-based index. This may be normalized by determining the offset ofeach call return address relative to its module base address as follows,such as when address space layout randomization (ASLR) is used:

-   -   First frame (0) is in ntdll.dll. 0_ip is the instruction pointer        return address, 0_mod_base is the module base.        Offset=0_ip−0_mod_base=53690    -   Second frame (1) is in kernel32.dll. Offset=102101    -   Third frame (2) is in advapi32.dll. Offset=108908

Each monitoring agent 104 preferably provides its snapshots 108 to asnapshot processor 112, such as via a computer network 114.

Snapshot processor 112 receives snapshots 108 from the variousmonitoring agents 104 for multiple events detected on the computers ingroup 100, normalizes their address information as described above, ifnecessary, and stores the snapshots 108 in a snapshot repository 116.Let S be a population of snapshots from group 100 for the same type ofevent detected for the same software application 106, and let Stk be aninstance of a call stack in S. [Stk|k] is defined as the first k framesof Stk. A scoring module 118 calculates, for all possible values of[Stk|k] in S, the number N([Stk|k]) as the number of computers in group100 that reported a snapshot that includes the same [Stk|k]configuration for the same type of event. Scoring module 118 calculatesthe probability of encountering a particular [Stk|k] configuration amongall the call stacks in S as:P([Stk|k])=N([Stk|k])/Σ_([stk′|]) N([Stk|k])where the summation is over all possible unique values of [stk′|k] in S(i.e., all possible unique configurations of the first k stack framesseen in group 100). Given the Shannon entropy of a discrete randomvariable P:Ent(P)=−Σp*log(p)scoring module 118 calculates the entropy for a given k as:Ent(k)=−Σ_([stk|k]) P([stk|k])*log(P([stk|k]))where the summation is over all possible unique values of [stk|k] in S(i.e., all possible unique configurations of the first k stack framesseen in group 100).

Scoring module 118 calculates a stack frame rarity score for a givencall stack configuration including its first k stack frames as:r(stk,k)=−log(P([stk|k]))−Ent(k)Scoring module 118 preferably calculates stack frame rarity scores for agiven call stack configuration for multiple, and preferably all,possible values of k available in the associated snapshot. Thus, forexample, where a snapshot of a call stack includes 10 stack frames, astack frame rarity score is calculated for the call stack when itincludes its first stack frame only, and a stack frame rarity score iscalculated for the call stack when it includes its first two stackframes, and so on up to 10 stack frames. Scoring module 118 thenpreferably calculates an overall call stack rarity score of a given callstack configuration as:R(stk)=max_(k) r(stk,k)where R(stk) is the maximal stack frame rarity score value of r(stk,k)seen after calculating it for multiple values of k.

Scoring module 118 may be additionally configured to calculate the valueArg max_(k) r(stk,k), denoting the “best” k value (i.e., fewest stackframes) that results in the maximum rarity score. Thus, where differentvalues of k result in the same maximum value, the lowest k of these ispreferably used. Scoring module 118 may be additionally configured tocalculate a representative stack prefix value using Stk and its best kvalue, which is simply the first best-k stack frames of Stk.

An anomaly detector 120 is configured to identify anomalous behavioramong any of the computers in group 100, based at least in part on thecall stack rarity score, optionally together with the best k valueand/or representative stack prefix of any of its snapshots. In oneembodiment, the computers in group 100 are ranked according to theircall stack rarity scores for snapshots of the same type of eventdetected for the same software application 106, and optionally by theirbest k values, where call stacks which have relatively high call stackrarity scores, such as in the top 5%, and, optionally, relatively lowbest k values as well, such as in the lowest 5%, are deemed anomalous.In another embodiment, any of the snapshots of the computers in group100 for a given type of event detected for the same software application106 is deemed anomalous if its representative stack prefix was notpreviously found in the snapshots reported by the computers in group100. In another embodiment, machine learning algorithms, such asalgorithms for the classification of malware invocations or clusteringof malware invocations, may be applied to the call stack rarity scores,best k values, and/or representative stack prefix of the snapshots insnapshot repository 116, optionally in addition to other known metricsrecorded for the computers in group 100, to determine that a call stackconfiguration of a given snapshot 108 is anomalous or otherwise identifyanomalous behavior among the computers in group 100.

Anomaly detector 120 preferably sends a notification to the monitoringagent 104 of any of the computers in group 100 that exhibits anomalousbehavior as described above, and/or to a user or administrator of thecomputer. Monitoring agent 104 is preferably configured to perform oneor more predefined computer-security-related remediation actions inresponse to receiving such a notification. The remediation actions may,for example, include terminating the execution of software application106, and/or providing a computer-security-related notification reportingthe anomalous behavior, such as to a user or administrator of thecomputer. Any of the above notifications may include the name ofsoftware application 106, the call stack rarity score, best k value, andrepresentative stack prefix of the associated call stack, or anycombination thereof.

Any of the elements shown in FIG. 1 are preferably implemented incomputer hardware in computer hardware and/or in computer softwareembodied in a non-transitory, computer-readable medium in accordancewith conventional techniques.

Reference is now made to FIGS. 2A and 2B which, taken together, is asimplified flowchart illustration of an exemplary method of operation ofthe system of FIG. 1, operative in accordance with an embodiment of theinvention. In the method of FIGS. 2A and 2B, instances of the samesoftware application are monitored on multiple computers having the sameoperating system (step 200). Snapshots of the software application callstack are recorded when the same type of predefined event is detected inconnection with the software application (step 202), such as a createprocess event. For multiple call stack configurations, and for multiplestack frame depths of the call stack configurations, the number ofcomputers that recorded the same call stack configuration at the samecall stack depth (i.e., first k frames) is determined (step 204).Probabilities of encountering particular call stack configurations atvarious call stack depths are determined (step 206). Stack frameentropies at various call stack depths are determined based on theirassociated stack frame probabilities (step 208). Stack frame rarityscores of particular call stack configurations at various call stackdepths are determined based on their associated stack frame entropies inaccordance with a predefined rarity function (step 210). The maximumstack frame rarity score is used as the overall call stack rarity score(step 212). The shallowest stack frame depth (i.e., fewest stack frames)of a particular call stack that produces its maximum stack frame rarityscore denotes its “best” k value (step 214). A representative stackprefix value of a particular call stack is calculated based on its bestk value (step 216).

Anomalous behavior among any of the computers is identified based atleast in part on the call stack rarity score, optionally together withthe best k value and/or representative stack prefix of any of thesnapshots (step 218). In one embodiment, the computers are rankedaccording to their call stack rarity scores for snapshots of the sametype of event detected for the same software application, and optionallyby their best k values, where call stacks which have relatively highrarity scores, such as in the top 5%, and, optionally, relatively lowbest k values as well, such as in the lowest 5%, are deemed anomalous(step 220). In another embodiment, any of the snapshots of the computersfor a given type of event detected for the same software application isdeemed anomalous if its representative stack prefix was not previouslyfound in the snapshots reported by the computers (step 222). In anotherembodiment, machine learning algorithms, such as algorithms for theclassification of malware invocations or clustering of malwareinvocations, may be applied to the call stack rarity scores, best kvalues, and/or representative stack prefix of the snapshots, optionallyin addition to other known metrics recorded for the computers, todetermine that a call stack configuration of a given snapshot isanomalous or otherwise identify anomalous behavior among the computers(step 224). A notification is sent to any of the computers that exhibitsanomalous behavior and/or to a user or administrator of the computer(step 226). One or more predefined computer-security-related remediationactions are performed in response to receiving the notification (step228), such as terminating the execution of the software application,and/or providing a computer-security-related notification reporting theanomalous behavior, such as to a user or administrator of the computer.Any of the above notifications may include the name of softwareapplication 106, the call stack rarity score, best k value, andrepresentative stack prefix of the associated call stack, or anycombination thereof.

Referring now to FIG. 3, block diagram 300 illustrates an exemplaryhardware implementation of a computing system in accordance with whichone or more components/methodologies of the invention (e.g.,components/methodologies described in the context of FIGS. 1, 2A, and2B) may be implemented, according to an embodiment of the invention.

As shown, the techniques for controlling access to at least one resourcemay be implemented in accordance with a processor 310, a memory 312, I/Odevices 314, and a network interface 316, coupled via a computer bus 318or alternate connection arrangement.

It is to be appreciated that the term “processor” as used herein isintended to include any processing device, such as, for example, onethat includes a CPU (central processing unit) and/or other processingcircuitry. It is also to be understood that the term “processor” mayrefer to more than one processing device and that various elementsassociated with a processing device may be shared by other processingdevices.

The term “memory” as used herein is intended to include memoryassociated with a processor or CPU, such as, for example, RAM, ROM, afixed memory device (e.g., hard drive), a removable memory device (e.g.,diskette), flash memory, etc. Such memory may be considered a computerreadable storage medium.

In addition, the phrase “input/output devices” or “I/O devices” as usedherein is intended to include, for example, one or more input devices(e.g., keyboard, mouse, scanner, etc.) for entering data to theprocessing unit, and/or one or more output devices (e.g., speaker,display, printer, etc.) for presenting results associated with theprocessing unit.

The descriptions of the various embodiments of the invention have beenpresented for purposes of illustration, but are not intended to beexhaustive or limited to the embodiments disclosed. Many modificationsand variations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed is:
 1. A computer anomaly detection and softwareexecution management method comprising: monitoring, on each of aplurality of computers running the same operating system, the executionof a software application on each of the computers, wherein the softwareapplication is executed in at least one instance on each of thecomputers; determining stack frame probabilities of encountering variousconfigurations of multiple call stacks at various call stack depths,wherein the call stacks are associated with the instances of thesoftware application on the computers, and wherein multiple snapshots ofthe call stacks are recorded on the computers responsive to detecting apredefined event in connection with the software application;determining stack frame entropies of various configurations of the callstacks at various call stack depths based on their associated stackframe probabilities; determining stack frame rarity scores of variousconfigurations of the call stacks at various call stack depths based ontheir associated stack frame entropies in accordance with a predefinedrarity function; determining a call stack rarity score of any givenconfiguration of the call stacks as the maximum stack frame rarity scoreof the given configuration; detecting an anomaly associated with anygiven one of the computers wherein any of the snapshots recorded on thegiven computer is of a call stack whose call stack rarity score meets apredefined anomaly condition, and terminating any of the executioninstances of the software application on any of the computers with whichthe anomaly is associated.
 2. The method of claim 1 and furthercomprising: recording the snapshots on the computers.
 3. The method ofclaim 1 wherein the detecting comprises ranking the computers accordingto their associated call stack rarity scores of their associatedsnapshots for the predefined event detected for the softwareapplication.
 4. The method of claim 1 wherein the detecting comprisesdetecting wherein the predefined anomaly condition is that arepresentative stack prefix based on the fewest stack frames associatedwith the call stack rarity score was not previously found in thesnapshots.
 5. The method of claim 1 wherein the detecting comprisesapplying a machine learning algorithm to the call stack rarity scores todetermine that any call stack configuration of any of the snapshots isanomalous.
 6. The method of claim 1 wherein the detecting comprisesapplying a machine learning algorithm to representative stack prefixesbased on the fewest stack frames associated with any of the call stackrarity scores to determine that any call stack configuration of any ofthe snapshots is anomalous.